Balanced Population Stochastic Variational Inference

Latent variable models are important tools to infer the underlying structure of a set of data. When we condition on observed data in Bayesian inference, we implicitly assume that the modeling assumptions are true and that the data can be considered a representative draw from the model. However, realistic data rarely agrees with these modeling assumptions. Especially when the observed data is highly imbalanced, inference will commonly result in redundant latent structures representing highly populated data clusters and miss the information contained in the scarce clusters. In this work, we propose a principled and scalable way to handle imbalanced data. We call our approach Balanced Population Stochastic Variational Inference (BP-SVI). Building on SVI, we use the Determinantal Point Process (DPP) to draw diversified mini-batches from the observed data set under the assumption that the observed data are an imbalanced realization of a true population. We show that this results in a more representative latent representation. From a theoretical side, our approach can be considered an instance of population Bayesian methods which were recently proposed for Bayesian inference on streams. While BP-SVI is applicable to a broad class of latent variable models, here we demonstrate how to use BP-SVI on Latent Dirichlet Allocation (LDA) as an example. Experiments are performed on both synthetic data and real-life news data showing clear improved performance on latent structure learning.